Toward Crowdsourced User Studies for Software Evaluation

This work-in-progress paper describes a vision, i.e., that of fast and reliable software user experience studies conducted with the help from the crowd. Commonly, user studies are controlled in-lab activities that require the instruction, monitoring, interviewing and compensation of a number of participants that are typically hard to recruit. The goal of this work is to study which user study methods can instead be crowdsourced to generic audiences to enable the conduct of user studies without the need for expensive lab experiments. The challenge is understanding how to conduct crowdsourced studies without giving up too many of the guarantees in-lab settings are able to provide. User studies are experimental and observational research methods for the measurement of an artifact’s properties as perceived by its users (we specifically focus on software artifacts, such as Web applications). They are, for instance, used to evaluate the strengths and weaknesses of different visualization techniques, to understand if theoretical principles hold in practical settings, to measure if requirements are met by a given software design, or to validate and test usability. Over the last decades, user studies have increasingly found their way into software engineering practice, and today it is almost impossible to find successful applications that do not consider the perception of their users. Facebook and Google, for example, can rely on an unprecedented user base to test new features on the fly and to adjust them according to observed performance or preferences. The problem is that not everybody has access to such a user base, e.g., because the own application has only a small target user group or because the application is still under development. Streamlining the necessary methods, involving the crowd, and providing user study support as a service while keeping study outputs reliable can thus make user studies significantly more accessible, to the benefit of everybody. The focus of this work is on how to crowdsource different user study methods conceptually and technically, i.e., on how to design effective tasks for user studies, gather and analyze data, guarantee quality and achieve representativeness. The question which method suits which research question is outside of its scope.